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I Abstract. After presenting the broad context of authority sharing, we 

outline how introducing more natural interaction in the design of the 
^ ' ground operator interface of UV systems should help in allowing a single 

O . operator to manage the complexity of his/her task. Introducing new 

■ modalities is one one of the means in the realization of our vision of next- 

generation GOT. A more fundamental aspect resides in the interaction 
manager which should help balance the workload of the operator between 
mission and interaction, notably by applying a multi-strategy approach 
^ to generation and interpretation. 

. We intend to apply these principles to the context of the Smaart pro- 

lyl ' totype, and in this perspective, we illustrate how to characterize the 

, workload associated with a particular operational situation. 
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CO ■ Unmanned Vehicle (UV) Systems will considerably evolve within the next two 

decades. Actually, in the current generation of UV Systems, several ground op- 
erators operate a single vehicle with limited autonomous capabilities. Whereas, 
in the next generation of UV Systems, a ground operator will have to super- 
vise a system of several cooperating vehicles performing a joint mission, i.e. a 
I Multi- Agent System (MAS) [10,12]. In order to enable mission control, vehi- 

cles autonomy will increase [7] and will require new forms of Human-system 
interaction. 

■ In this context, we have developed a prototype multi- UV ground control 

H I station (Smaart) that allows an operator to supervise the surveillance of a sim- 

ulated strategic airbase by a swarm of rotary- wing UVs [13]. In this system, the 
autonomous behavior of the UVs is generated by the means of a dual digital 
pheromone algorithm (bio-inspired approach), and although we obtained inter- 
esting results, the operator-system interaction is rather basic: place beacons in 
the environment, dispatch UVs towards these beacons, select mode of informa- 
tion display, etc. Despite being technically an authority sharing system, Smaart 
is only the first step in the development of a full-fledged authority sharing con- 
trol system for swarms of UVs. Two main challenges have to be faced in order 
to design such an efficient and realistic UV Systems: 



1. obviating the negative side effects of automation: workload mitigation, loss 
of situation awareness, complacency, skill degradation, etc [17]; 



2. decreasing the cognitive load induced for the ground operator. Operating 
UV systems is highly complex. Obviously, shifting to UV Systems with sev- 
eral vehicles will makes mission and vehicles control more complex [6]. In 
addition, even though increasing vehicles' autonomy aims at decreasing the 
cognitive load induced by mission control for ground operators, workload 
mitigation may lead to even higher workload [17,6]. 

While using an UV System's interface, the ground operator is at least engaged 
within two activities: mission control and interaction. However, all in all, the 
great majority of study focus on mission control. In this paper, we claim that 
interaction design must be considered as a field of research by itself. In this 
perspective, as [25, 9] we claim that there is a need to enhance the naturalness 
of ground operator interface rather than only improving mission realization and 
control. But as soon as an interface provides "natural" input and/or output 
devices, non-understandings and misunderstandings may occur. Then, in order 
to design efficient UV Systems, the problem of robustness of the interaction has 
to be handled. 

Based on recent advances in Pragmatics and in Human-Computer Interface 
[4,22,18,19], we present a collaborative view of interaction dedicated to UV 
Systems. Considering interaction as a collaborative activity while designing an 
interface enhances its robustness [20], opens the door to managing the global 
workload of the operator through a balance effect between mission load and 
interaction load [16]. 

2 Perspective on Authority Sharing 

This section presents the backdrop of our research on authority sharing for un- 
manned systems control. The basic requirement for an authority sharing system 
is to provide several distinct operating modes to accomplish a given task or 
function. Fig. 1 illustrates how we represent the different operating modes for 
three tasks, decomposed along John Boyd's OODA loop [23]. In this example, 
the system has only one operating mode for the Observe stage of Tl, but two 
alternative modes for Orient and Decide, and three modes for Act. 




Fig. 1. Representation of the operating modes for three tasks (Tl, T2, T3). 



Each operating mode gives specific authority and responsibilities to the op- 
erator and the system. For example, the Act stage of the landing task of an UV 
could provide three modes: (a) full manual i.e. the operator tele-operates the 
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Fig. 2. Authority sharing concepts in single system-single operator interactions. 



UV; (b) full auto i.e. the vehicle lands automatically with no possible interven- 
tion of the operator; and (c) auto with veto, where the operator can disconnect 
the auto-pilot and handle the landing. 

As we see on Fig. 2, in order to control the system at time t, an operating 
mode must be activated for each task and stage in OODA (filled cells on Fig. 1). 
Therefore, the main question is "which operating mode has to be selected at time 
t?" This corresponds to a second level of authority sharing i.e. the authority to 
assign responsibilities (select operating modes) and must be implemented as 
some sort of decision process. Indeed, in a full-fledged authority-sharing system, 
the operators and the machines could have diff'erent preferences concerning which 
operating modes should be selected. 

We can consider that the operators and the system's internal representations 
can be broadly decomposed along the same three categories: 

— models and representations of the situation: his corresponds to the repre- 
sentation of objects in the system's environment, knowledge about its laws, 
properties, etc.: this can be seen as a "world model"; 

— models and representations of the system: his corresponds to the current 
state of the system, its known capabilities, predictions about its evolution, 
etc.; 

— models and representations of the operator: similarly this represents the state 
of the operator, his/her abilities, performances, etc. 



If one combines these categories with the two kind of actors (machine and 
Humans), we obtain six distinct fields of research relevant to the development 
of authority sharing systems. Without being exhaustive, one can identify: 

— work on human situational awareness (representation of the situation on the 
operator's side); 

— work on trust in automation (representation of the system on the operator's 

side); 

— cognitive and physiological modeling (representation of the operator on the 
system's side). 

System's and operator's representations are not only fed by observation of 
the situation (as illustrated on Fig. 2), but also through interaction between Hu- 
man and machine. Man-machine interaction can happen on several media (from 
classic mouse & keyboard, joystick to advanced haptic interfaces or dialogue) 
but whatever the; chosen media, it should tend to facilitate the convergence be- 
tween the respective representations of Humans and machines involved. Again, 
we consider here another field of research by itself. And one can note that an 
efficient interaction will decrease the difficulty of the final decision-making pro- 
cess as converging representations on the Human and machine's sides lead to 
converging preferences on which operating modes to select. 

3 Naturalness 

In the current generation of UV Systems, Ground Operator Interface (GOI) 
is a traditional Graphical User Interface, such as [11]. These are based on in- 
put/output modalities such as drop-down menu or push button, with a con- 
strained interaction language, providing quantitative spatial information and 
interaction, etc. This interaction language is similar to the low-level command 
language for vehicles, with quantitative spatial information for example. Thus, 
GOI naturalness is poor. However, current works aims at enhancing the natural- 
ness of interface [3]. That is integrating a less-constrained interaction, at least 
a single natural modality as input [16,9] (such as gesture, spoken or written 
language) or output [9] (such as speech, haptic display), multi- modality, flexible 
interaction [12], providing qualitative spatial information and interaction etc. 

As soon as naturalness is introduced within the GOI, non-understandings - 
due to vagueness, ambiguity or underspecification - have to be managed. Inter- 
active management of non-understandings follows from the collaborative nature 
of interaction (grounding) and requires a new GOI component: an interaction 
manager [20] . Such an approach has been used within The WITAS project [14] 
as well as within the GeoDialogue Project [2], which relates to Geographical 
Information Systems. 

Enhancing GOI naturalness has various benefits for UV Systems. First, a 
more natural interaction between an interface and its user generally enhances 
the eflSciency of interaction i.e. it reduces the cognitive load induced by the 
interaction for the user as well as interaction time. For example, a data entry 



function based on vocal keyword recognition may require a single vocal utterance 
in the next generation of GOI, while it may require over twenty separate manual 
actions in the current generation of GOI. 

Moreover, natural display modality (typically, haptic display) also aims at 
making up for the "sensory isolation" of the ground operator [9]. Operator's 
sensory isolation is due to the fact that he is generally not collocated in the same 
physical space than vehicles [8,21]. This leads to lack of situation awareness, 
among others negative effects. 

Supervisory control of UV systems mainly involves spatial cognition and ref- 
erence to vehicles, landmarks, waypoints, etc. To the extend that human beings 
performs these tasks using qualitative information and interact through gesture 
and verbal communication, GOI naturalness should focus on spatial information 
and interaction. Nonetheless our goal is not to transform the GOI in a fully 
natural interface. But, as soon as naturalness is introduced within an interface, 
side effects have to be carefully considered. In particular, the GOI must contain 
new functionalities: being a semantic bridge between operators and vehicles, and 
handling of non-understandings. 
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Fig. 3. Ground operator interface: a semantic bridge. 



First, considering "natural" input device {i.e. corresponding to a control 
command from the ground operator to a vehicle), there is a mismatch between 
the "natural" command provided by the operator and the "operational" com- 
mand that a vehicle can accept. Then, the ground operator interface must be a 
semantic bridge, that converts the perceived message in a representation which 
is suitable for the addressee. That is to say that following the perception of an 
input on a control input device and following its interpretation, GOI also has to 
convert the understood control command before transmitting it to the proper 
vehicle(s). As shown on Fig. 3: 



— the front part the GOI has to detect input on a control input device and has 
to transmit the raw message to the interpretation module; 

— the interpretation modulo identifies the intend meaning of the operator: the 
kind of control command and the intended objects, the two vehicles which 
are the recipients of the message and the intended pre-defined zone; 

— the conversion module has to: 

1. translate operator command into the language which is proper for the 
addressee; 

2. process the; underspecified elements, such as path planning. The conver- 
sion module may lack necessary information, such as the first waypoint. 
In this case, a completion request has to be send to the ground operator 
in order to complete the command conversion. 



Second, as soon as an interface provides semi-constrained interaction, qualita- 
tive spatial interaction [2], natural (multi-)modality [22], then non-understandings 
may occur. Non-understanding is commonly set apart misunderstanding. In a 
misunderstanding, the addressee succeeds in communicative act's interpretation, 
whereas in a non-understanding he fails. But, in a misunderstanding, addressee's 
interpretation is incorrect. For example, mishearing may lead to misunderstand- 
ing. 

Handling non-understandings is necessary for the GOI as soon as an in- 
put control command cannot be transmitted to vehicles. Human beings handle 
non understandings by interactively refining their understanding until a point 
of intelligibility is reached. This process is called "grounding" [4]. In order to 
design grounding, interaction has to be viewed as a collaborative process and 
an interaction manager has to be integrated within the GOI. For more details, 
the interested reader may refer to [20]. There is a common misconception that 
non-understandings are considered as "communicative errors" one may tend to 
avoid, as well as understanding refinement. Although, either for Human-Human 
Interaction [5] or for Human-Computer Interaction [15], non-understandings and 
their management process present lots of advantages. For example, feedbacks are 
cue that enable interaction partners to be aware of the level of understanding of 
each other. Through interaction refinement each interaction partner maintains 
an accurate representation of the other and this enhances the efficiency of future 
interactions. 

More generally, enhancing interaction design may lead to positive side effects 

enhancing mission control. If one takes the example of gesture, as explained in the 
previous section, enabling the ground operator to interact using gesture aims at 
making efficient references. Actually, gestures facilitate the maintenance of spa- 
tial representations in working memory [24]. Therefore, gestures may contribute 
to maintaining the situation awareness that enables an efficient supervisory con- 
trol by the operator. 



4 Balancing Mission &: Interaction 



While using an UV System's interface, the ground operator is at least engaged 
within two activities: mission control and interaction. This is the general case of 
all goal-oriented interaction (or dialogue): 

Dialogues, therefore, divide into two planes of activity [4]- On one plane, 

people create dialogue in service of the basic joint activities they are en- 
gaged in making dinner, dealing with the emergency, operating the ship. 
On a second plane, they manage the dialogue itself - deciding who speaks 
when, establishing that an utterance has been understood, etc. These two 
planes are not independent, for problems in the dialogue may have their 
source in the joint activity the dialogue is in service of, and vice versa. 
Still, in this view, basic joint activities are primary, and dialogue is cre- 
ated to manage them. [1] 

Interaction is defined by each dialog partner's goals to understand each other, 
i.e. words to reach a certain degree of intelligibility, sufficient for the current 
purpose. The crucial points here are that : 

1. perfect understanding is not required, the level of understanding required is 
directed by the basic activity {i.e. the mission) and the situational context 
{e.g. time pressure); 

2. as ground operator's cognitive load is "divided" between the cognitive loads 
induced by each activity, the interaction's complexity must vary depending 
on the complexity involved by the mission, as defined by Mouloua and al. [16]. 
For example, as time pressure rises, the cognitive load induced by the mission 
increases. The cognitive load required by the interaction should decrease in 
order to carry through the mission. 

In the perspective of adapting adapt the grounding process to the specific 
case of the supervision of multiple UVs, one can note that a GOI is also an 
interaction support for a team, therefore similar to interfaces dedicated to Com- 
puter Supported Cooperative Work. However, the team include humans (ground 
operators) and machines (vehicles). The GOI is also an interaction partner of 
ground operators. They interact for decision support tasks, non-understanding 
management and interface manipulation. 

In addition, our study aims at designing interaction in order to take advan- 
tage of the mutual dependancy between mission control and interaction. More 
precisely, we propose adaptive interaction design in order to obtain the prop- 
erties defined by Mouloua and al. [16,20]. The point is to obviate the negative 
side effects of automation, through balanced workload. We claim that adapting 
interaction design in regards human factors such as operator's workload, trust or 
performance must have positive side effects on these human factors, cf. Figure 4. 

The GOI we arc developing will integrate an Interaction Manager compo- 
nent, responsible for choosing adequate strategies for (1) generation and (2) 
interpretation: 
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1 . for a given information or interaction to be potentially send to the operator, 
deciding whether to send it or not and choosing a modality and formulation; 

2. trying to understand interactions emanating from the operator, given the 
current context and grounding information. 

Several redundant strategies will have to be developed for the various possible 
interactions permitted by the GOI. For a given interaction, some strategies will 
put the burden of interaction on the GOI (disambiguation, acknowledgement, 
eic.) while others will rest more "on the shoulders" of the operator. The rationale 
is that in situations of low "mission workload" , one is better off with giving 
more work the operator as it builds up the grounding all the while keeping 
him/her busy (fending off boredom and loss of attention). Conversely, in mission 
critical episodes, the GOI can assume a more active role in interaction and let 
the operator focus on the mission (and still be robust thanks to the grounding 
constructed earlier). 

5 Situation Cueing for the GOI of a Swarm of UVs 

In addition to designing different strategies for generation and interaction, we 
have to give to the interaction manager some means to evaluate the current state 
of the mission (and therefore make an estimation of the associated workload) in 
order to choose strategies. In this section, we illustrate how - in the perspective 
of the extension of the Smaart system (see [13] in this volume) - we intend to 
discriminate four categories of mission states and their associated workloads. 

5.1 Description 

The Smaart system allows patrol and intercept operations for a dozen of rotary- 
wing UAVs supervised by a single operator. 





Fig. 5. Illustration of levels of workload. Symbols and correspond to - respec- 
tively - patrolling and pursuing UVs. The symbols correspond to alarms. Thick 
dotted lines link alarms that are supposed to have been triggered by the same intruder, 
while greyed-out regions are search zones with adjustable parameters (center, direction 
and breadth). Recent alarms are indicated by an exclamation mark (!). 



Subfigure 5a illustrates the lowest level of workload: Routine Patrol. The 
UAVs perform their surveillance with a stable performance, every point on the 
airbase is scanned at an acceptable frequency. In this context, the task of the 
operator is purely of a supervisory nature. The parameters of the display are set 
to values known to be adequate for the current setup (number of UAVs, area, 
threat level, etc. see [13] in this volume). The operator has a good appreciation 
of the reliability of the algorithm (good) and knows that anomalies are rare. 

Due to its local nature, the algorithm that the UAVs use to perform their 
coordinated patrol can misbehave in some configurations.'^ In such cases, the 
operator can detect the anomaly and take actions by dispatching some UAVs 
manually to compensate for the anomaly. In this context he/she has to closely 
supervise the execution of these actions, judge their effectiveness, all the while 

^ For example, if for some reasons an "islet" appear in the digital pheromone space, the 
gradient-following UAVs will never reach it, and consequently, a dark spot will appear 
and worsen. This local processing on the part of the UAVs has some interesting 
properties, but also has the consequence that a modification in a part of the airbase 
{e.g. creation of a no-fly zone) will take some time to be propagated to the rest of 
the environment though the digital pheromone. 



continuing his/her global supervising activity of the patrol on the whole airbase. 
One can detect such a workload level {Patrol with Anomaly) by the action of 
the operator on an UAV (Subfigure 5b). 

The two next workload levels are characterized by the presence of alarms. 
The number of alarms in recent time allows to distinguish low threat Alarm 
(possible false alarm, Subfigure 5c) from emergency situation (multiple alarms, 
coordinated Intrusion, Subfigure 5d). In this last situation, the general surveil- 
lance of the airbase is largely jeopardized, as (1) many UAVs are used to pursue 
the intruders in specific regions, therefore depleting the patrolling vehicles. And, 
(2) the attention of the operator is largely focused on the intrusions. 

5.2 Characterization 

Table 1 sums up the characterization of the operational situations we intend to 
implement in Smaart. 



1 


No activity on the part of the operator (no command sent to the UAVs) 


2 


At least a command sent to a UAV in the last few minutes 


3 


One alarm in the Icist few minutes 


4 


Several alarms in the last few minutes 



Table 1. Characterization of the four levels of mission workload in Smaart. 



Based on these criterions, the interaction manager is able to compute a dis- 
crete mission workload level at every moment: either (1) by storing every events 
(operator action toward UAVs or alarms) and matching with the criterions of 
table 1, or (2) by updating a continuous workload level by the combination of 
fixed additive values associated to alarms and orders with a discount temporal 
factor (see Figure 6). With the latter option, the continuous level is compared 
to pre-defined thresholds to obtain discrete levels. 

6 Conclusion &: Perspectives 

In the broad context of authority sharing, we have outlined how introducing more 
natural interaction in the design of the ground operator interface of UV systems 
should help in allowing a single operator to manage the complexity of his/her 
task. Introducing new modalities is one one of the means in the realization of 
our vision of next-generation GOI. A more fundamental aspect resides in the 
interaction manager which should help balance the workload of the operator 
between mission and interaction, notably by applying a multi-strategy approach 
to generation and interpretation. 

We intend to apply these principles to the context of the Smaart prototype, 
and in this perspective, we have illustrated how to characterize the workload 
associated with a particular operational situation. 
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Fig. 6. Illustration of the computation of the four levels of mission workload. 
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